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METHOD AND COMPUTER SYSTEM FOR CONTROLLING ACCESS BY 
APPLICATIONS TO THIS AND OTHER COMPUTER SYSTEMS 

FIELD OF THE INVENTION 

The present invention generally relates to computer systems and associated methods and 
more particularly to a system and associated method for protecting computer systems from 
unauthorized access by certain applications, 

DESCRIPTION OF THE RELATED ART 

With the increasing sophistication of computer networks and the ability of the public to 
access private networks (e.g., through the Internet) there have been a number of recent systems 
and computer programs developed for protecting private networks from being improperly 
accessed by the public. For example, it is common to use sophisticated passwords and other 
security devices to prevent unauthorized access to a private network. However, the 
sophistication of the password and the ability to break passwords is an ever escalating 
"competition" which quickly renders password security applications obsolete. Therefore, 
password security applications require constant updating and maintenance. Further, all of the 
users must have a password to access the system, which limits the effectiveness of password 
security for Web servers. 

Another such security measure is known as a firewall. In one type of firewall system, the 
users transfer files to a firewall host, and then log into the firewall and subsequently transfer the 
file to the desired external party. Using a firewall prevents users from making a direct 
connection outside the private network and also prevents the public from reaching the private 
network because information can only be transferred by the firewall host. Other systems allow 
certain external users to "tunnel" through the firewall to allow direct access to the private 
network. Such systems contain high levels of security to allow the tunneling process to take 
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place. However, firewalls and tunnels through firewalls are very cumbersome because the 
private network and the public need to deal with the intermediary firewall. This is a slow and 
cumbersome process and adds to the cost and complexity of the system. 

Other systems use a proxy server to hide the identity of the private network. In such a 
system, communications are passed through the proxy server, which appears to the public to be 
the private network. However, only the proxy knows the true identity and location of the private 
network and the proxy does not allow the public access to the private network. As with the 
firewall system, the proxy system is cumbersome, expensive and requires constant maintenance. 

Therefore, there is a need for a low-overhead system which limits access by an untrusted 
computer system to a private trusted computer system, but which simultaneously allows the 
trusted system full access to the untrusted system. 

SUMMARY OF THE INVENTION 

According to the present invention, application execution contexts within an untrusted 
computer system are classified as either trusted or untrusted based on distinctive application 
execution context names. A human administrator of the untrusted system assigns these 
distinctive application execution context names. The applications in the untrusted system cannot 
change the names of their own execution contexts (or preferably any other execution contexts in 
the untrusted computer system). 

The untrusted application execution contexts are fenced off from certain parts of the 
untrusted computer system such that the untrusted application execution contexts cannot 
interrogate or change critical system data of the untrusted computer system. 
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Only applications on the untrusted computer system which execute in trusted execution 
contexts can initiate a connection with a trusted computer system. The trusted computer system 
can initiate connections with any execution context on the untrusted computer system. Only the 
untrusted application execution contexts on the untrusted system can initiate connections with an 
external computer system. The connections originating on the external system can terminate 
only at the untrusted system and can terminate only at untrusted execution contexts therein. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects, aspects and advantages will be better understood from 
the following detailed description of preferred embodiments of the invention with reference to 
the drawings, in which: 

Figure 1 is a schematic diagram of the present invention. 

Figure 2 is a flow diagram illustrating operation of the present invention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS OF THE INVENTION 

Referring now to the drawings, and more particularly to Figure 1, a first embodiment of 
the present invention is illustrated in schematic form. The trusted system 10 shown in Figure 1 
comprises a private network to which users are permitted access through some form of security 
such as passwords, physical restriction, etc. The external system 12 shown in Figure 1 represents 
computer networks owned by the public, such as the Internet and other private networks. One 
major concern is that the external system 12 may try to reach the trusted computer system 10 to 
misappropriate data or destroy the trusted system 10. 
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To prevent this, the invention positions an untrusted computer system 1 1 between the 
trusted computer system 10 and the external system 12. The untrusted system 11 includes an 
operating system 14 which controls many devices such as storage device 15 (e.g., a disk drive). 
The untrusted system exists to host data or run applications which must be made available to 
external system 12. 

A key feature of the invention is that the untrusted system includes some applications that 
run in trusted application execution contexts 13 and other applications that run in untrusted 
application execution contexts 16. With the invention, the untrusted application execution 
contexts 16 cannot initiate communications with the trusted system 10. However, trusted 
application execution contexts 13 can initiate connections with the trusted system 10. Programs 
running on trusted system 10 can initiate connections to any context on untrusted system 11. 

The invention distinguishes between trusted application execution contexts 13 and 
untrusted application execution contexts 16 according to distinctive application execution 
context names. A human administrator of system 1 1 with operating system 14 assigns each 
application execution context a distinctive name, and an application running in untrusted system 
1 1 cannot tamper with the name of any execution context (i.e., its own or any others). The 
creation of the distinctive names and the classification of the names as trusted or untrusted is 
performed with a control in the operating system 14. In one example, the prefix for the trusted 
names (user ids) is a specifically-chosen alphanumeric character string, (i.e., the administrator of 
the operating system might specify that all names beginning with the characters "SFS" are 
trusted). 

Certain applications running on behalf of users on external system 12 are given names by 
a human administrator of the operating system 14 which indicate that they are running in 
untrusted execution contexts 16. Other applications running on behalf of trusted system 10 are 
given other names by the human administrator of operating system 14 which indicate that they 
are running in trusted execution contexts 13. 
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Any application which is running in a context that has been assigned a "trusted" name by 
the system administrator executes as a trusted application. These "trusted" contexts (user ids) are 
unable to communicate with the external system 12. By segregating the application contexts into 
trusted and untrusted, allowing only the trusted application contexts 13 to initiate connections 
with the trusted system 10, and allowing the external network 12 to access only the untrusted 
contexts 16, security is achieved. 

In a preferred embodiment two distinct communications protocols are used to distinguish 
between requests from the external system 12 and the trusted system 10. Requests from the 
external system 12 use one protocol (e.g., the TCP/IP protocol). Communications to/from the 
trusted system 10 use a different protocol (e.g., the APPC communications protocol). In this 
embodiment, the operating system 14 and the storage 15 are set up such that they cannot be 
manipulated or administered via the TCP/IP protocol (that is, TELNET access is disabled). 
Therefore, the operating system 14 and storage 15 cannot be compromised by users on external 
system 12. However, distinct communications protocols are not required to make the invention 
operable. So long as the operating system can determine the origin of a request through some 
means (e.g., by noting the TCP/IP interface that the request came through) the invention will 
operate properly. 

Therefore, the invention provides a method for connecting two environments where there 
is a trusted side 10 and an untrusted side 1 1 so that the trusted side 10 can have access to all of 
the data (i.e., the trusted network 10 itself and the untrusted system 11), but the external system 
12 can access only a subset of such data (i.e., the data made available through the applications 
running in untrusted contexts 16). 

For example, the invention is very useful where a business has an Internet web page 
(which would reside on the untrusted system 1 1) yet still wants to have the Internet web data (in 
addition to all of its internal business data) accessible by employees who are on the trusted 
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system 10. In the foregoing situation, the business with the trusted system 10 would need full 
control over access to the trusted system 10, through conventional security measures (e.g., 
passwords, physical isolation, etc.). 

The security of the invention is achieved by the inventive ability of the operating system 
14 to decide which application execution contexts are permitted to initiate connections to the 
trusted system 10. The communications software on the operating system 14 is programmed so 
that connection attempts with the trusted system 10 cannot originate from the untrusted 
application execution contexts 16. As mentioned above, the operating system 14 lets connection 
attempts to the trusted system 10 originate only from trusted execution contexts 13. The 
untrusted computer system 1 1 has a communication software stack and operating system whose 
controls and configuration parameters are adequately fenced from potentially untrustable 
applications. In other words, the operating system and communication stack are installed and 
configured such that untrusted contexts do not have access to their configuration information and 
are not authorized to control them. This is typically done with file permissions but each 
operating system will have different means. 

With the system described above, if a user on the trusted system 10 attempts to connect to 
the untrusted system 1 1 to access some data, the user would be communicating with a trusted 
application execution context 13 and the connection would be allowed (and data would flow over 
the connection). However, if an external user 12 (accessing untrusted system 1 1 via an untrusted 
application execution context 16) tries to initiate a connection with the trusted system 10, the 
connection is rejected by the operating system 14. Therefore, the invention allows connections 
between the trusted system 10 and the untrusted system 1 1 to be initiated only from the trusted 
system 10, or from an application running in a trusted execution context 13. 

Figure 2 is a flow chart illustrating such operation of the present invention. In step 20, 
operating system 14 determines the name of the context originating a communication. This 
determination is made by retrieving the contents of the "context name" field of the data structure 
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(a.k.a. "control block") describing the initiating context. Next, in step 21, the operating system 
14 compares the context name to a list of trusted names (maintained in primary memory or in 
storage 15) or to a convention for trusted names (such as a particular prefix indicating 
trustworthiness). Then, operating system 14 determines whether the application execution 
context that is originating communication is trusted or untrusted. If the application execution 
context is untrusted, the context is not permitted to initiate communication with trusted system 
10, as shown in step 22. However, if the application execution context is trusted, the context is 
permitted to initiate communication with the trusted system 10, as shown in step 23. 

As an example, programming implementing steps 20-23 of the present invention could be 
added to an existing IBM VM/ESA Version 2 Release 4 operating system to yield operating 
system 14. In this scenario, untrusted system 1 1 would run the VM/ESA operating system 
augmented with the present invention and trusted system 10 would run the standard VM/ESA 
operating system without such augmentation. A networking connection between these two 
VM/ESA operating systems could be an existing VM/ESA Inter-System Facility for 
Communication (ISFC) link. When the augmented VM/ESA operating system 14 in untrusted 
system 1 1 determines that a connection is not permitted with trusted system 10 because the 
connection request originates from an untrusted execution context 16, operating system 14 does 
not let the untrusted context 16 use ISFC. However, if operating system 14 determines that a 
connection is permitted with trusted system 10 because the connection request originates from a 
trusted execution context 13, operating system 14 lets the trusted execution context 13 use ISFC 
to initiate Advanced Program-to-Program Communication (APPC) connection to the trusted 
system 10. The APPC protocol is described in IBM publication SC24-5760, "VM/ESA CP 
Programming Services", and other references. 
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By allowing trusted application execution contexts 13 in the untrusted system 1 1 to 
initiate communications with the trusted system 10, various VM/ESA features requiring such 
capability, such as the Shared File System (SFS), are easily enabled. The present invention 
would permit SFS to initiate the connection to trusted system 10 because SFS would be running 
5 in a trusted execution context 13. 

With the present invention, system administration and overhead are reduced because only 
one operating system on two (untrusted and trusted) computer systems are required (even though 
one is modified). As explained above in the "Description of the Related Art", in order to 
conventionally attain the results produced by the present invention, several servers running 

10 multiple conventional operating systems would have to be utilized to ensure that users operating 
the untrusted system were denied access to the trusted system. Each of these servers would 

O require more code (which slows the system down) and additional administrator time to set up the 

gi servers. Further, such a conventional system would be substantially more complicated to 

~ maintain than the present invention. 

|jj> As noted above, applications running in untrusted execution contexts 16 cannot 

U interrogate or update critical or sensitive system 1 1 data, such as a password file, the set of 
$ enrolled users, the list of trusted execution context names or the record of the name prefix that 
O distinguishes trusted contexts from untrusted ones. This can be accomplished based on file 
S permission and access schemes specific to the type of operating system 14. For example, when 
20 operating system 14 is an augmented VM/ESA operating system as explained above, applications 
running in untrusted execution contexts 16 are denied knowledge of and access to the disk 
volumes containing the CP directory, the TCP/IP communication stack configuration files, and 
the startup script containing the commands that activate ISFC filtering and define the trusted 
context name prefix. The CP directory contains the list of known users (that is, execution 
25 contexts) and their passwords. The TCP/IP communication stack configuration files list the 
names of the contexts that are allowed to communicate with external system 12. The startup 
script, run automatically by VM/ESA at startup time, contains privileged commands that activate 
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the filtering of ISFC activity based on context name and define the name prefix that identifies 
trusted execution contexts. 

While the invention has been described in terms of preferred embodiments, those skilled 
in the art will recognize that the invention can be practiced with modification within the spirit 
and scope of the appended claims. 
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CLAIMS 



What is claimed is: 



1 1 . A method for controlling access to a computer system comprising: 

2 classifying applications running on an untrusted computer system as running in one of a 

3 trusted application execution context and an untrusted application execution context; and 

4 preventing an application on said untrusted computer system from initiating a connection 

5 with a trusted computer system unless said untrusted computer system is running said 

6 application in said trusted application execution context. 

01 2. The method in claim 1, wherein said trusted computer system can initiate connections 
S£ with any execution context on said untrusted computer system. 

fll 3. The method in claim 1, wherein only said untrusted application execution contexts on 
%2 said untrusted system can initiate connections with said external computer system. 

a 4. The method in claim 1, wherein said applications are classified as having said trusted 

application execution contexts and said untrusted application execution contexts based on 
3 distinctive application execution context names. 

1 5. The method in claim 4, wherein a human administrator of said untrusted system assigns 

2 said distinctive application execution context names. 

1 6. The method in claim 4, wherein said applications cannot change the names of respective 

2 execution contexts in which said applications are running. 
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1 7. The method in claim 4, wherein said applications cannot change the name of any 

2 execution context in said untrusted computer system. 

1 8. The method in claim 1, wherein connections originating on said external system can 

2 terminate only at said untrusted system and only at said untrusted execution contexts 

3 therein. 

1 9. The method in claim 1, wherein said untrusted application execution contexts are fenced 

2 off from said untrusted computer system such that said untrusted application execution 

3 application contexts cannot interrogate or change critical system data of said untrusted 

4 computer system. 

CI 10. A method for controlling access to a trusted computer system comprising: 

^ determining a name of an execution context of an application running on an untrusted 

J| system; 

~4 determining whether said execution context is trusted or untrusted based on said name; 

fjf) if said execution context is trusted, permitting said application to initiate a connection 
with said trusted system, and 

7 if said execution context is untrusted, preventing said application from initiating a 

8 connection with said trusted computer system. 

1 11. The method in claim 10, wherein said trusted computer system can initiate connections 

2 with any execution context on said untrusted computer system. 
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1 12. The method in claim 10, wherein only said untrusted application execution contexts on 

2 said untrusted system can initiate connections with an external computer system. 

1 13. The method in claim 10, wherein said execution context name was previously assigned 

2 by a human administrator. 

1 14. The method in claim 10, wherein there are a plurality of applications running on said 

2 untrusted computer system, one of said applications having a trusted execution context 

3 and another of said applications having an untrusted execution context. 

1 15. The method in claim 14, wherein said applications cannot change names of the respective 

2 execution contexts in which said applications are running. 

gl 16. The method in claim 14, wherein said applications cannot change the name of any 
execution context in said untrusted computer system. 

Ill 17. The method in claim 10, wherein connections originating on an external system can 
%Q, terminate only at said untrusted system and only at said untrusted execution contexts 

therein. 

3 18. The method in claim 10, wherein said untrusted application execution contexts are fenced 

2 off from said untrusted computer system such that said untrusted application execution 

3 application contexts cannot interrogate or change critical system data of said untrusted 

4 computer system. 

1 19. A program storage device readable by machine, tangibly embodying a program of 

2 instructions executable by the machine to perform a method for controlling access to a 

3 computer system, said method comprising: 
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4 classifying applications running on an untrusted computer system as running in one of a 

5 trusted application execution context and an untrusted application execution context; and 

6 preventing an application on said untrusted computer system from initiating a connection 

7 with a trusted computer system unless said untrusted computer system is running said 

8 application in said trusted application execution context. 

1 20. The program storage device in claim 19, wherein said trusted computer system can 

2 initiate connections with any execution context on said untrusted computer system. 

1 21. The program storage device in claim 19, wherein only said untrusted application 

2 execution contexts on said untrusted system can initiate connections with said external 
G| computer system. 

S| 22. The program storage device in claim 19, wherein said applications are classified as 

J| having said trusted application execution contexts and said untrusted application 

!13 execution contexts based on distinctive application execution context names. 

23. The program storage device in claim 22, wherein a human administrator of said untrusted 

C2 system assigns said distinctive application execution context names. 

1 24. The program storage device in claim 22, wherein said applications cannot change names 

2 of respective execution contexts in which said applications are running. 

1 25. The method in claim 22, wherein said applications cannot change the name of any 

2 execution context in said untrusted computer system. 
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1 26. The program storage device in claim 19, wherein connections originating on said external 

2 system can terminate only at said untrusted system only at said untrusted execution 

3 contexts therein. 

1 27. The program storage device in claim 19, wherein said untrusted application execution 

2 contexts are fenced off from said untrusted computer system such that said untrusted 

3 application execution contexts cannot interrogate or change critical system data of said 

4 untrusted computer system. 

1 28. A system for controlling access to a network comprising: 

2 a trusted computer system; 

p| an untrusted computer system connected to said trusted computer system and to an 

external computer system, 

fjji wherein said untrusted system includes applications classified as having trusted 

application execution contexts and untrusted application execution contexts, and 

£:! wherein only said trusted application execution contexts can initiate connections with 

5$ said trusted computer system. 

1 29. The system in claim 28, wherein said trusted computer system can initiate connections 

2 with any execution context on said untrusted computer system. 

1 30. The system in claim 28, wherein only said untrusted application execution contexts on 

2 said untrusted system can initiate connections with said external computer system. 
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1 31. The system in claim 28, wherein said applications are classified as having said trusted 

2 application execution contexts and said untrusted application execution contexts based on 

3 distinctive application execution context names. 

1 32. The system in claim 31, wherein a human administrator of said untrusted system assigns 

2 said distinctive application execution context names. 

1 33. The system in claim 31, wherein said applications cannot change the names of respective 

2 execution contexts in which said applications are running. 

1 34. The method in claim 31, wherein said applications cannot change the name of any 

2 execution context in said untrusted computer system. 



gi 35. The system in claim 28, wherein connections originating on said external system can 
^2 terminate only at said untrusted system and only at said untrusted execution contexts 

yb therein. 

J4 36. The system in claim 28, wherein said untrusted application execution contexts are fenced 
J| off from said untrusted computer system such that said untrusted application execution 

C!3 application contexts cannot interrogate or change critical system data of said untrusted 

3 computer system. 
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METHOD AND COMPUTER SYSTEM FOR CONTROLLING ACCESS BY 
APPLICATIONS TO THIS AND OTHER COMPUTER SYSTEMS 



ABSTRACT 



Application execution contexts within an untrusted computer system are classified as 
trusted or untrusted based on respective names assigned to the execution contexts. If an 
application runs in an untrusted execution context, an operating system within the untrusted 
computer system prevents the application from initiating a connection with a trusted computer 
system and accessing sensitive parts of the untrusted computer system. If the application runs in 
a trusted execution context, the operating system permits the application to initiate a connection 
with the trusted computer system. 
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priority is claimed: 

Prior Foreign Application(s): 

N um ber Country Day/Month/Year Priority Claimed 

I hereby claim the benefit under Title 35, United States Code, § 120 of any United States application(s) listed below and, insofar as the subject matter 
of each of the claims of this application is not disclosed in the prior United States application in the manner provided by the first paragraph of Title 35 
United States Code, § 1 12, 1 acknowledge the duty to disclose material information as defined in Title 37, Code of Federal Regulations, § 1.56 which 
occurred between the filing date of the prior application and the national or PCT international filing date of this application: 

Prior U.S. Applications: 

Serial No. Filing Date Status 

I hereby declare that all statements made herein of my own knowledge are true and that all statements made on information and belief are believed to 
be true; and further that these statements were made with the knowledge that willful false statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Tide 18 of the United States Code and that such willful false statements may jeopardize the validity of 
the application or any patent issued thereon. 

As a named inventor, I hereby appoint the following attorneys and/or agents to prosecute this application and transact all business in the Patent and 
Trademark Office connected therewith: David LArthur (Reg No. 29,604), Lawrence R. Fraley (Reg No. 26,885), John R. Pivmchny (Reg No. 43,00 1), 
Arthur J. Samodovitz (Reg No. 31,297), William H. Steinberg (Reg No. 28,540), and Frederick W. Gibb, III (Reg. No. 37,629) and Sean M. McGinn 
(Reg No. 34,386). 

Send all correspondence to: McGinn & Gibb, P.C„ 1701 Clarendon Boulevard, Suite 100, Arlington, Virginia 22209.Customer No. 21254 

Telephone calls should be directed to McGinn & Gibb, P.C. at (703) 294-6699. 



re: ^ X- 5 v2*~~~ Date: ^a^U^ l^Zcoa 



(1) Inventor: John S. Roman 

Signature: 

Residence: 386A Great Road, Apt. 6, Acton, Massachusetts 01720 
Citizenship: United States of America 
Post Office Address: Same as Residence 



Inventor: Brian K. Wade 



Signature: . 



Residence: 9 Highland Drive, Apalachin, NY 13732 
Citizenship: United States of America 
Post Office Address: Same as Residence 
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